In today's world, customer service has become a crucial part of any business, and companies are turning towards AI-powered chatbots to handle customer queries and support requests.

ChatGPT is an advanced chatbot that uses cutting-edge "Generative Pre-trained Transformer" technology, to understand and respond to human interactions. By applying "Natural Language Processing" techniques, ChatGPT provides relevant and helpful responses to a wide range of questions and queries.

ChatGPT is a versatile tool that can do it all, from answering questions and assisting with tasks like composing emails and essays to writing code and simulating entire chat rooms.

Nevertheless, with the prevalence of powerful technology accessible to the public, there are individuals misusing ChatGPT for malicious purposes. There is now proof on the dark web that certain individuals have taken advantage of ChatGPT to produce malicious content, even with anti-abuse restrictions in place to deter such illegitimate requests.

Let’s take a deeper dive into the world of ChatGPT and explore its limitations and risks, which are also related to the limitations of AI in general.

You can read more about AI in our previous article “7 AI Myths that You Need to Stop Believing”.


ChatGPT Limitations

1- ChatGPT can generate responses based solely on the data it has been trained on, which means it may not always provide accurate or relevant information.

2- Additionally, since ChatGPT is pre-trained on data from the internet, it may not have the contextual understanding of specific industries or businesses.

3- ChatGPT can sometimes generate biased responses based on the training data it has been exposed to, which can be a problem for sensitive topics like politics and social issues.

4- One of the primary restrictions of ChatGPT is its data limitation, which extends only up to 2021. Therefore, the chatbot lacks knowledge about any recent events or news that may have taken place since then.

5- ChatGPT is not capable of emotional intelligence, so it cannot empathize with customers or understand the nuances of human emotion.

6- According to OpenAI, ChatGPT tends to produce answers that sound plausible but could be sometimes incorrect or nonsensical. This phenomenon is known as "hallucination," and it is a common behavior exhibited by large language models.
 

*OpenAI

But… the question remains: is ChatGPT dangerous?

To answer this question, we resorted to ChatGPT, and here’s what it had to say about itself:

“As an AI language model, I am not inherently dangerous. I am a tool that can be used for a variety of purposes, both positive and negative. Like any technology, the way I am used depends on the intentions and actions of the people using me.

My purpose is to facilitate communication and provide information to the best of my ability while following ethical and legal guidelines. However, there is always the possibility that someone could misuse my responses or use them to harm others…”

In short, ChatGPT lacks emotional or physical capabilities, is intended to engage in text-based conversations with humans, and is not considered harmful to people.

ChatGPT Risks

Attackers can use ChatGPT to deceive and target users and their devices. That being said, let’s explore some of the risky business that ChatGPT is capable of:

1- Lack of morals

ChatGPT does not have a moral or ethical framework of its own, as it is designed to generate responses based on statistical patterns in data used to train it, without any consideration to the underlying moral principles involved.

This can be problematic for humans where they may expect a moral or ethical judgment, and where ChatGPT may provide a response that conflicts with a human's own morals.


2- Ability to deceive people

There is a possibility that someone could use OpenAI's technology to create a fraudulent customer service chatbot, that may appear to be a legitimate representative, to deceive people. This can include the use of common phrases, jargon, and industry-specific language, all of which help to build the illusion of a real human agent on the other end of the conversation.

Moreover, ChatGPT can be used to create fake customer service pages or websites that mimic legitimate companies, where the chatbot can extract sensitive information such as login credentials or personal details from unsuspecting customers.

*Tom Brown et al.

3- Generating spam and phishing emails

An attacker could train ChatGPT on a dataset of legitimate emails from a specific company or industry, making it easier than ever to create convincing spam and phishing emails, mimicking the style of such reputable companies.

For example, an attacker could use ChatGPT to generate an email that appears to be from a bank, requesting that the recipient click a link to update their account information. If the recipient falls for the scam and clicks the link, they may unwittingly provide their personal or financial information to the attacker, potentially leading to financial loss or identity theft.


4- Writing malware

It is possible that someone could use ChatGPT to generate text-based instructions or code that could be used to create malware. When asked to write dangerous malware, many times ChatGPT complied and delivered.

Despite OpenAI’s efforts in improving the chatbot's restraints to prevent malicious or abusive activities, cybercriminals can still manipulate the program by using specific words or phrases that allow them to bypass restrictions. It is possible to get the model to do what you want it to do in various ways.

5- Discrimination based on gender and race

While ChatGPT has anti-discrimination measures in place, it can still perpetuate biases and discrimination in certain situations. For example, it may inadvertently use discriminatory language in programs that determine credit limits or salaries.

This might happen because the data used to train ChatGPT contained language reflecting societal biases or stereotypes about certain groups of people.

Manually filtering data to remove explicitly racist content can be a challenging and resource-intensive task for humans. This is one of the reasons why AI models like ChatGPT are trained on large datasets of text, which may contain implicit biases and stereotypes that are difficult to detect and remove. AI models like ChatGPT are only as good as the data they are trained on, ongoing efforts are therefore needed to ensure they are used ethically and responsibly.


In conclusion, new technological innovations like ChatGPT have the potential to profoundly shape societies, but they are also vulnerable to exploitation by malicious actors.

As with any powerful tool, it is important to use ChatGPT and other AI language models with a critical eye and to carefully consider the implications of their responses. By doing so, we can help ensure that these technologies are used ethically and responsibly and that they contribute to a better, more equitable world for all.

You can’t simply do what the machine told you to do ;)